Skip to content

Fix: Prevent duplicate edge_job insertions for deferrable tasks in EdgeExecutor (#53610)#53927

Merged
jscheffl merged 5 commits intoapache:mainfrom
Prasanth345:fix-53610-edgeexecutor-deferrable-duplicate-job
Aug 4, 2025
Merged

Fix: Prevent duplicate edge_job insertions for deferrable tasks in EdgeExecutor (#53610)#53927
jscheffl merged 5 commits intoapache:mainfrom
Prasanth345:fix-53610-edgeexecutor-deferrable-duplicate-job

Conversation

@Prasanth345
Copy link
Contributor

@Prasanth345 Prasanth345 commented Jul 30, 2025

Overview

This PR fixes a crash in the Apache Airflow EdgeExecutor caused by duplicate entries in the edge_job table when handling deferrable tasks like ExternalTaskSensor(mode="reschedule").

Root Cause

When a deferrable task times out and retries, both execute_async() and queue_workload() attempt to insert the same job into the edge_job table again, violating uniqueness constraints and causing the scheduler to crash.

Solution

Instead of blindly inserting, both methods now:

  • Check if a matching job exists (dag_id, task_id, run_id, map_index, try_number)
  • If it exists → Update the job to reflect the latest state
  • If not → Insert a new record

This preserves job queue integrity and ensures tasks are picked up correctly by the edge worker.

Changes Made

  • Patched execute_async() and queue_workload() in:
    airflow/providers/edge3/src/airflow/providers/edge3/executors/edge_executor.py

Related Issue

Closes #53610

@Prasanth345 Prasanth345 requested a review from jscheffl as a code owner July 30, 2025 12:54
@boring-cyborg
Copy link

boring-cyborg bot commented Jul 30, 2025

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@boring-cyborg boring-cyborg bot added area:providers provider:edge Edge Executor / Worker (AIP-69) / edge3 labels Jul 30, 2025
jscheffl
jscheffl reviewed Jul 30, 2025
View reviewed changes
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VEry good! Thanks for the fix! I can confirm this resolved the problem!

To be merged, can you please also extend the existing use cases in providers/edge3/tests/unit/edge3/executors/test_edge_executor.py to prevent a future regression?

@Prasanth345
Copy link
Contributor Author

Prasanth345 commented Jul 31, 2025

@jscheffl Added test coverage under test_edge_executor.py to cover the updated behavior and prevent future regression.

@jscheffl
Copy link
Contributor

jscheffl commented Jul 31, 2025

@jscheffl Added test coverage under test_edge_executor.py to cover the updated behavior and prevent future regression.

Thanks!
Just a small glitch of asserts in tests in Airflow 2.10/2.11 - can you fix this? The command is validated and the mocked test fails in your pytest. Then I think it is good to be merged.

@Prasanth345
Copy link
Contributor Author

Prasanth345 commented Aug 1, 2025

@jscheffl Thanks for the feedback!

I’ve updated the code to conditionally call validate_airflow_tasks_run_command using a hasattr check. This ensures compatibility with Airflow 2.10 and 2.11, where the method may not be present.
All tests are now passing across supported versions.

jscheffl
jscheffl reviewed Aug 3, 2025
View reviewed changes
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"almost" good to merge but some nit about log formatting/commented code. Then I think it is good to merge.

jscheffl
jscheffl approved these changes Aug 4, 2025
View reviewed changes
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the multiple iterations for the fix and staying tuned until now! Finally all is green and looking good! LGTM! Thanks!

@jscheffl jscheffl merged commit c2059b9 into apache:main Aug 4, 2025
71 checks passed
@boring-cyborg
Copy link

boring-cyborg bot commented Aug 4, 2025

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@eladkal eladkal mentioned this pull request Aug 7, 2025
Closed
40 tasks
ferruzzi pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Aug 7, 2025
@Prasanth345 @ferruzzi
Fix: Prevent duplicate edge_job insertions for deferrable tasks in Ed…
4f62e4b 
…geExecutor (apache#53610) (apache#53927)

* Fix: Handle duplicate entries in EdgeExecutor for deferrable tasks

* Add test cases to validate EdgeExecutor job update and worker sync behavior

* Fix: Ensure execute_async handles command validation safely across Airflow versions

* Fix test to use valid Airflow task command format

* Clean up: Remove unused debug logs from EdgeExecutor

---------

Co-authored-by: lakshminarayana.kumbha <lakshminarayana.kumbha@zemosolabs.com>
fweilun pushed a commit to fweilun/airflow that referenced this pull request Aug 11, 2025
@Prasanth345 @fweilun
Fix: Prevent duplicate edge_job insertions for deferrable tasks in Ed…
edc5931 
…geExecutor (apache#53610) (apache#53927)

* Fix: Handle duplicate entries in EdgeExecutor for deferrable tasks

* Add test cases to validate EdgeExecutor job update and worker sync behavior

* Fix: Ensure execute_async handles command validation safely across Airflow versions

* Fix test to use valid Airflow task command format

* Clean up: Remove unused debug logs from EdgeExecutor

---------

Co-authored-by: lakshminarayana.kumbha <lakshminarayana.kumbha@zemosolabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jscheffl jscheffl jscheffl approved these changes

Assignees

No one assigned

Labels

area:providers provider:edge Edge Executor / Worker (AIP-69) / edge3

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Task fails to execute in EdgeExecutor if it is deferred and re-scheduled

2 participants

@Prasanth345 @jscheffl